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ABSTRACT 

For nost tests administered vith tiie limits soie 
examinees complete all items vhile others do not. It is often useful 
to knov vhat the distribution of items completed vould be if the 
number of items on the test vere much larger. It may also be of 
interest to estimate the correlation between working speed, as 
measured by the number of items completed, and the total score. The 
method described assumes that working speed and total score are 
bivariate normally distributed in the population of interest and 
allows estimation of the means and standard deviations for items 
completed and total score, and the correlation between items 
completed and total score given a theoretically infinite number of 
items in the same time limit. (Author) 
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Estimating Total Score and Item Completion Statistics 
When All Ex:iminees Do Not Finish 



Richard Reilly 
Educational Testing Service 

There are situations in which it is usefful to know the extent 
to which examinees' working speed is related to their total score* 
If no examinees finish the test than the correlation between the 
number of items completed (c) and total score (x) should provide a 
good estimate of this relationship* In many cases, however, a 
significant proportion of examinees finish the test and all receive 
the same score on c , thereby making the usual correlation 
coefficient inappropriate* Ideally, a much longer test (long enough 
so that no examinees finished) composed of additional parallel items 
could be readmlnistered with the same time limits to a new saiq>le 
and the desired estimate obtained* Because this approach is rarely 
feasible an alternative method is proposed which can be used to 
estimate r^^ , and at the same time provide the following additional 
parameter estimate? for a theoretical test consisting of a large 
number of parallel items: 

c » the mean number of items completed* 

s^ the standard deviation of items completed* 

s » the total score standard deviation* 

X 

X » the total score mean* 
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Estimating the Mean and Standard Deviation of Items Completed 

The method involves the assumptions that items completed and 
total score have a bivariate normal distribution and that examinee 
working speed would not be affected by an additional set of parallel 
items. Figure 1 illustrates the observed and theoretical distribution 
of items completed tor a test which a large proportion of examinees 



Insert Figure 1 About Here 

did not finish. The distribution has been divided into two segments: 
A, the actual distribution of items completed; and B, the theoretical 
distribution for examinees who finished the test. The two. segments 
£.r£ d'^vided by a line at c^ which represents the actual number of 
ite'iis in the test. 

To obtain the estimate s^ the follow*.ng relationships are 
employed: 



(1) 



1. 
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z e d 
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where 

p is the pi^oportion of examinees not finishing the 
n ^ 

test. 

is the unit normal deviated corresponding to p^^ • 

is the first moment of the truncated unit normal*^ 
distribution with terminus z 

c 

is the second moment of the truncated unit normal 
distribution. 

is th3 standard deviation of the truncated normal 
distribution. 

s^ is the estimated standard deviation of items completed 
^ for those not finishing the test. 

s is the estimate of the standard deviation of items 
completed for the conqplete theoretical distribution 
(i.e., finishers and non-finishers with an infinite 
number of additional parallel items) . 

Once s^ has been obtained it becomes possible to obtain c , the 

estimated mean number of items completed. Actually it is possible 

to obtain c in two different ways as follows: 



c- = c - z s ; or (6) 
1 o c c 



C2 = -'-«c4 



where c is k-1 for a k-item test, and c^ is the mean number 
o 

of items completed for examinees not finishing the test. If the 
distribution is normal these two estimates should agree fairly closely, 
if the two estimates differ widely the distribution of non-finishers 
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should be examined using a method similar to that proposed by 
Donlon (1975) • It may be that the normality assumption does not 
hold. 

Table 1 shows the mean and standard deviations for truncated 

normal distributions with truncation points corresponding to different 

proportions of non-finishers, p^ . Table 1 is presented for 

illustrative purposes rather than as a complete table. Values of 

y and a can be computed for exact values of p with the help 

n 

of a complete table of the normal distribution. 

Insert Table 1 About Here 



Estimating the Correlation Between Items Completed and Total Score 
Given s^ , the followirg formula can be used to obtain r^^ 



s r 
c xc 



xc 



^s^ (r^ )^ + (s^)^ (1 (r" )h 
c xc c xc 



(8) 



where 



r « the correlation between items completed and total 
score for non-finishers. 



Estimating the Mean and Standard Deviation of Total Score 

The mean of the total score distribution can be estimated as 
follows: 



X « x" - b(c"-c) 



(9) 
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where 

s 

b = r' -4 

XC 

c 

s"" = total score standard deviation for non-f Anishers, 
and 

x^ is the mean total score for non-finishers. 

The standard deviation of the total score can be estimated as 



^ " ^'x f 2 2 2 • 



The reader will recognize formulae (8) and (10) as variants of 
those presented by Gulliksen (1950, pp. 137-138) in his discussion 
of explicit selection. Variants of (8), (9), and (10) were derived 
by Cohen (1955) as maximum likelihood estimates of parameters of 
the bivariate normal distribution when one variable is restricted. 
Cohen (1950) also presented an alternative method of estimating the 
mean and standard deviation of a sample drawn from a truncated 
normal distribution. The method requires the computation of 



N 
n 

V, » Z (c - c )/N ; and (11) 
1 i 0 n 



N 

V- - Z (c - c '//N . (12) 
2 i 0 n 
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These values are then substituted it formulas provided by Cohen (1950) 
and an iterative procedure must then be used to arrive at estimates 
of the mean and standard deviation of c • Cohen's procedures are 
computationally more laborious than the previous method but have 
the advantage of providing maximum likelihood estimates of the mean 
and standard deviation* 

Example ; A new 30-item test of arithmetic is administered to 
I9OOO students. Total time allowed for the test is 20 minutes* The 
test is completed by 45Z of the students. The test developers are 
interested in adding more items to the test but would like to know 
what the relationship between working speed anc' total score is. They 
are also interested in knowing what the mean and standard deviation 
of items completed would be for a much longer test. Finally, they 
want some idea of the total score mean and standard deviation for a 
much longer test. 

In this particular case 550 students did not complete the test. 
The first row of Table 2 shows statistics based on the 550 non- 
finishers only. The second row of Table 2 shows estimates for the 
total sample which were obtained using the relationship given above. 

Insert Table 2 About Here 



Conment 

The methods presented here are baaed on assumptions which should 
be more widely investigated with empirical data. Good empirical 
validation of the utility of these assumptions and the methods presented 
here could be done with large samples of examinees and tests which no 
(or an insignificant percentage of) examinees finish. 
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Figure 1 

Illustration of Actual and Theoretical 
Distribution of Items Completed 
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Table 1 

Means and Standard Deviations for the Unit Normal Distribution 
Truncated at Points Corresponding to p 

n 







a' 


.05 


-2.06271 


.34400 


.10 


-1. 75498 


.40431 


.15 


-1.55439 


.43936 


.20 


-1.39981 


.47774 


.25 


-1.27111 


.50021 


.30 


-1.15898 


.51878 


.35 


-1.05828 


.54111 


.40 


-0.96586 


.55471 


.45 


-0.87957 


.58367 


.50 


-0.79788 


.60281 


.55 


-0.71965 


.62584 


.60 


-0.64390 


.64981 


.65 


-0.56984 


.67506 


.70 


-0.49670 


.70201 


.75 


-0.42370 


.73123 


.80 


-0.34995 


.76355 


.85 


-0.27430 


.80029 


.90 


-0.19500 


. 84385 


.95 


-0.10856 


.89980 



ERIC 



11 



-10- 




o 

CO 



CM 
H 

I 



U 

CO 

u 

00 



M 
O 

CO 
M 

4j 



o 

CO 
4J 



U 
CO 
H 



O 

CO 
4J 



4J 

CO 
H 



o 



I I 





0) 








u 








o 








u 






• 


CO 


O 


o 


Q 




o 


CM 


• 


H 


• 


• 


CO 


cd 


CO 














o 








H 







1 


-0 




0) 




u 


M 


le 


• 


a 


Q 


6 


• 




CO 


5 



o 



CO 



o 



CO 
CM 



O 
CM 



00 
CM 




ERIC 



12 



